Systems and methods for implementing retail processes based on machine-readable images and user gestures

ABSTRACT

Systems and methods for implementing retail processes based on machine-readable images and user gestures are disclosed. According to an aspect, a method includes capturing one or more images including a machine-readable image and a user hand gesture. The method also includes identifying the machine-readable image as being associated with a product. Further, the method includes determining whether the user hand gesture interacts with the machine-readable image in accordance with a predetermined gesture. The method also includes implementing a predetermined retail process in association with the product in response to determining that the user hand gesture interacts with the machine-readable image in accordance with the predetermined gesture.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation-in-part application of co-pending U.S. patentapplication Ser. No. 14/189,354, filed Feb. 25, 2014 and titled SYSTEMSAND METHODS FOR IMPLEMENTING RETAIL PROCESSES BASED ON MACHINE-READABLEIMAGES AND USER GESTURES, which claims the benefit of U.S. ProvisionalPatent Application No. 61/924,334, filed Jan. 7, 2014 and titled SYSTEMSAND METHODS FOR IMPLEMENTING RETAIL PROCESSES BASED ON MACHINE-READABLEIMAGES AND USER GESTURES; the contents of which are hereby incorporatedherein by reference in their entireties.

FIELD

The present subject matter relates to retail devices and equipment, andmore specifically, to implementation of retail processes based onmachine-readable images and user gestures.

BACKGROUND

In retail environments, such as grocery stores and other “brick andmortar” stores, retail personnel and customers interact with productsand other objects located in the retail environment. As an example, acustomer may carry a computing device, such as a smartphone or wearablecomputing device, and use the computing device for purchasing theproduct either in the store or through an online retail system. Thecomputing device may be operable to capture an image of a barcode of theproduct for identification of the product in a subsequent purchasetransaction. In this way, the computing device may be used to facilitatepurchase of the product while the user is located within a retailenvironment.

In another example, retail personnel may use a computing device forconducting store inventory. The computing device may be used to scanproduct barcodes. In this way, the products may be identified for thepurposes of inventorying the store.

In view of the foregoing, it is desirable to provide improved techniquesand devices for facilitating user interaction with products in a retailenvironment.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing summary, as well as the following detailed description ofvarious embodiments, is better understood when read in conjunction withthe appended drawings. For the purposes of illustration, there is shownin the drawings exemplary embodiments; however, the presently disclosedsubject matter is not limited to the specific methods andinstrumentalities disclosed. In the drawings:

FIG. 1 is a block diagram of an example system for implementing retailprocesses based on machine-readable-images and user gestures inaccordance with embodiments of the present subject matter;

FIG. 2 is a flow chart of an example method for implementing retailprocesses based on machine-readable images and user gestures inaccordance with embodiments of the present subject matter;

FIG. 3 is a screen display of an example image captured by a computingdevice in accordance with embodiments of the present subject matter;

FIG. 4 is a screen display of another example image captured by acomputing device in accordance with embodiments of the present subjectmatter;

FIG. 5 is a screen display of another example image captured by acomputing device in accordance with embodiments of the present subjectmatter; and

FIG. 6 is a screen display of another example image captured by acomputing device in accordance with embodiments of the present subjectmatter.

DETAILED DESCRIPTION

The presently disclosed subject matter is described with specificity tomeet statutory requirements. However, the description itself is notintended to limit the scope of this patent. Rather, the inventors havecontemplated that the claimed subject matter might also be embodied inother ways, to include different steps or elements similar to the onesdescribed in this document, in conjunction with other present or futuretechnologies. Moreover, although the term “step” may be used herein toconnote different aspects of methods employed, the term should not beinterpreted as implying any particular order among or between varioussteps herein disclosed unless and except when the order of individualsteps is explicitly described.

As referred to herein, the term “computing device” should be broadlyconstrued. It can include any type of device including hardware,software, firmware, the like, and combinations thereof. A computingdevice may include one or more processors and memory or other suitablenon-transitory, computer readable storage medium having computerreadable program code for implementing methods in accordance withembodiments of the present subject matter. A computing device may be,for example, retail equipment such as POS equipment. In another example,a computing device may be a server or other computer located within aretail environment and communicatively connected to other computingdevices (e.g., POS equipment or computers) for managing accounting,purchase transactions, and other processes within the retailenvironment. In another example, a computing device may be a mobilecomputing device such as, for example, but not limited to, a smartphone, a cell phone, a pager, a personal digital assistant (PDA), amobile computer with a smart phone client, or the like. In anotherexample, a computing device may be any type of wearable computer, suchas a computer with a head-mounted display (HMD). A computing device canalso include any type of conventional computer, for example, a laptopcomputer or a tablet computer. A typical mobile computing device is awireless data access-enabled device (e.g., an iPHONE® smart phone, aBLACKBERRY® smart phone, a NEXUS ONE™ smart phone, an iPAD® device, orthe like) that is capable of sending and receiving data in a wirelessmanner using protocols like the Internet Protocol, or IP, and thewireless application protocol, or WAP. This allows users to accessinformation via wireless devices, such as smart phones, mobile phones,pagers, two-way radios, communicators, and the like. Wireless dataaccess is supported by many wireless networks, including, but notlimited to, CDPD, CDMA, GSM, PDC, PHS, TDMA, FLEX, ReFLEX, iDEN, TETRA,DECT, DataTAC, Mobitex, EDGE and other 2G, 3G, 4G and LTE technologies,and it operates with many handheld device operating systems, such asPalmOS, EPOC, Windows CE, FLEXOS, OS/9, JavaOS, iOS and Android.Typically, these devices use graphical displays and can access theInternet (or other communications network) on so-called mini- ormicro-browsers, which are web browsers with small file sizes that canaccommodate the reduced memory constraints of wireless networks. In arepresentative embodiment, the mobile device is a cellular telephone orsmart phone that operates over GPRS (General Packet Radio Services),which is a data technology for GSM networks. In addition to aconventional voice communication, a given mobile device can communicatewith another such device via many different types of message transfertechniques, including SMS (short message service), enhanced SMS (EMS),multi-media message (MMS), email WAP, paging, or other known orlater-developed wireless data formats. Although many of the examplesprovided herein are implemented on smart phone, the examples maysimilarly be implemented on any suitable computing device, such as acomputer.

As referred to herein, the term “user interface” is generally a systemby which users interact with a computing device. A user interface caninclude an input for allowing users to manipulate a computing device,and can include an output for allowing the computing device to presentinformation and/or data, indicate the effects of the user'smanipulation, etc. An example of a user interface on a computing deviceincludes a graphical user interface (GUI) that allows users to interactwith programs or applications in more ways than typing. A GUI typicallycan offer display objects, and visual indicators, as opposed totext-based interfaces, typed command labels or text navigation torepresent information and actions available to a user. For example, auser interface can be a display window or display object, which isselectable by a user of a computing device for interaction. The displayobject can be displayed on a display screen of a computing device andcan be selected by and interacted with by a user using the userinterface. In an example, the display of the computing device can be atouch screen, which can display the display icon. The user can depressthe area of the display screen where the display icon is displayed forselecting the display icon. In another example, the user can use anyother suitable user interface of a computing device, such as a keypad,to select the display icon or display object. For example, the user canuse a track ball or arrow keys for moving a cursor to highlight andselect the display object.

As referred to herein, the term “machine-readable image” can be anyimage that can be recognized by a computing device. For example, amachine-readable image may be a barcode or any other opticalmachine-readable representation of data, such as a two-dimensional (2D)or three-dimensional (3D) code. In another example, a machine-readableimage may be a portion of a product or item that can be recognized by acomputing device for identifying the product or item. In anotherexample, a machine-readable image may be all or a portion of a label ofa product or item that can be recognized by a computing device foridentifying the product or item.

The presently disclosed subject matter is now described in more detail.For example, FIG. 1 illustrates a block diagram of an example system forimplementing retail processes based on machine-readable-images and usergestures in accordance with embodiments of the present subject matter.Referring to FIG. 1, the system includes a computing device 100including an image capture device 102 and a retail process manager 104.The computing device 100 may be a wearable computing device such as, butnot limited to, a computing device including a mechanism configured toenable wear of the computing device. In an example, the computing device100 may be a tablet computer. In another example, the computing device100 may be an optical head-mounted display device (e.g., the wearablecomputer referred to as Google Glass available from Google of MenloPark, Calif.). The computing device 100 may include a user interface106, such as a display (e.g., touchscreen display), a touchpad, and/orthe like. The image capture device 102 may be configured to capture oneor more images and/or video.

It is noted that in the examples provided herein, a single computingdevice (e.g., computing device 100) is described as implementing thedisclosed function. However, it should be understood that the functionsdisclosed herein may alternatively be implemented by multiple computingdevices. The computing devices may be communicatively connected via asuitable network. In an example, the functions disclosed herein may beimplemented by a system having multiple image capture devices, such as aretail store system having multiple cameras. In this example, themultiple image capture devices may capture images or video of productsand user gestures relative to the products. The captured images or videomay be used for implementing processes in accordance with embodimentsdisclosed herein.

The computing device 100 may include a network interface 108 configuredto communicate with another computing device via a BLUETOOTH®communications connection or another suitable communications connection.The network interface 108 may communicate with another computing device,such as a server 110, via a network 112 (e.g., a local area network or awide area network).

The computing device 100 may be configured to use a display of the userinterface 106 to implement augmented reality functions. For example, theimage capture device 102 may capture and display images on the display.Further, the computing device 100 may simultaneously control thedisplay, along with the captured images displayed in real time,augmented elements of the captured images. For example, the displayimages may also include information relating to objects or areas in thecaptured images.

The retail process manager 104 may be implemented by hardware, software,firmware, or combinations thereof. For example, the retail processmanager 104 may include one or more processors and memory 114 includinginstructions for implementation by the processor(s). The retail processmanager 104 may be connected to the image capture device 102, the userinterface 106, and the network interface 108 for implementation of thefunctionality disclosed herein.

FIG. 2 illustrates a flow chart of an example method for implementingretail processes based on machine-readable images and user gestures inaccordance with embodiments of the present subject matter. The method ofFIG. 2 is described by example as being implemented by the retailprocess manager 104 of the computing device 100 shown in FIG. 1,although it should be understood that the method may be implemented byany suitable computing device(s). Further, reference is made to theexample image shown in FIG. 3 and described in further detail herein,although it should be understood that any one or more other images maybe used in accordance with embodiments of the present subject matter.

Referring to FIG. 2, the method includes capturing 200 one or moreimages including a machine-readable image and a user gesture. Forexample, the retail process manager 104 may control the image capturedevice 102 to capture one or more images. In an example, the imagecapture device 102 may capture a still image or video of a barcode.Example barcodes include, but are not limited to, two-dimensionalbarcodes and matrix barcodes. The video may be captured in real time.Further, the captured image or video data may be stored in the memory114. As an example, the retail process manager 104 may capture a userhand gesture including a finger and hand of a user pointing to andrecognizing the user hand gesture as a tapping gesture with respect tothe machine-readable image. Alternatively, the retail process manager104 may recognize that the hand may be making a point gesture and movingin a swiping motion across the recognized barcode in a particulardirection.

The retail process manager 104 may control a display (e.g., the displayon a tablet computer or an optical head-mounted display device) of theuser interface 106 to display the captured image or video. The image orvideo may be displayed in real time. For example, the user may be storeassociate using the computing device to implement retail operations in aretail store. In an example, the store associate may be conductinginventory in the store.

FIG. 3 illustrates a screen display of an example image captured by acomputing device in accordance with embodiments of the present subjectmatter. The image one frame of a video captured of a side of a product300 having a barcode 302 printed on a side thereof. Further, thecaptured video may include a hand 304 of a user of the computing device100 moving in a direction indicated by the direction arrow 306.

Referring again to FIG. 2, the method includes identifying 202 themachine-readable image as being associated with a product. Continuingthe aforementioned example, the retail process manager 104 may beconfigured to receive one or more captured images of the barcode 302shown in FIG. 3. The retail process manager 104 may be configured tointerpret the numerical digits (or other data) that the barcode 302represents and associate the numerical digits with an identifier for theproduct 300. Based on the product identifier, the retail process manager104 may be configured to request from the server 110 or otherwise obtaininformation about the product 300 based on the identifier.

The method of FIG. 2 includes determining 204 whether the user gestureinteracts with the machine-readable image in accordance with apredetermined gesture. Continuing the aforementioned example, the retailprocess manager 104 may recognize that the hand 304 is the hand of auser of the computing device 100. Further, the retail process manager104 may recognize that the hand 304 is making a point gesture and movingin a swiping motion across the recognized barcode 302 in the direction306. Alternatively, for example, two fingers on different hands may cometogether about the outside of the barcode 302 area to indicate acommand. The command may represent a selection of the product 300associated with the barcode 302. The user gesture may be deemed by theretail process manager 104 to be in accordance with a predeterminedgesture for interacting with the barcode 302. Although a swiping gestureand a point gesture is described as being recognized in this example, itshould be appreciated that any other suitable gesture captured in one ormore images may be recognized as being indicative of an interaction witha machine-readable image in accordance with the present subject matter.

The method of FIG. 2 includes implementing 206 a predetermined retailprocess in association with the product in response to determining thatthe user gesture interacts with the machine-readable image in accordancewith the predetermined gesture. Continuing the aforementioned example,the retail process manager 104 may add the product to an inventory list,a purchase list, or any other list in response to determining that thehand 304 is making a point gesture and moving in a swiping motion acrossthe recognized barcode 302 in the direction 306. This functionality maybe used, for example, for inventorying items within a retail store. Forexample, the product may be added to the inventory list afterrecognition of the user gesture interacting with the barcode 302.

In another example, the retail process manager 104 may conduct apurchase transaction for purchase of the product 300 in response todetermining that the hand 304 is making a point gesture and moving in aswiping motion across the recognized barcode 302 in the direction 306.This functionality may be used, for example, for user by a customer inpurchasing the product by adding the product to a cart for subsequentcheckout and purchase.

In accordance with embodiments of the present subject matter, the retailprocess manager 104 may control the display to display an augmentedreality interface together with the display of one or more images.Referring again to FIG. 3, the augmented reality interface may includethe boxes in the figure containing the phrases “Buy Product,” “ProductInformation,” and “Product Reviews”. The user may interact with theboxes to initiate a command. For example, the “Buy Product” box may beinteracted with to initiate purchase of the product or add to product toa cart for subsequent purchase. The “Product Information” box may beinteracted with to request additional information about the product, andthe information may be displayed via the augmented reality interface.The “Product Reviews” box may be interacted with to request reviewinformation about the product, and the review information may bedisplayed via the augmented reality interface.

In accordance with embodiments of the present subject matter, the retailprocess manager 104 may determine a position of the user gesture in theimage(s) and controlling the display to display an identifier foridentifying the position of the user gesture. For example, the tip ofthe finger of the hand 304 shown in FIG. 3 may be augmented via asuitable augmented reality technique to include an indicia to show thelocation of the fingertip. In an example, the fingertip may be augmentedwith a colored dot such that the user may see on the display that thecomputing device 100 recognizes the finger and/or hand 304.

In an example scenario, a customer may be in a retail store and maycarry the computing device 100. Further, the customer may pick up anitem, such as the product 300 shown in FIG. 3. Further, the customer mayorient the product 300 such that the barcode 302 faces the image capturedevice 102. Further, the customer can point his or her finger at the oneend of the barcode, and the retail process manager 104 can recognize thebarcode 302 and augment the words “barcode located, swipe to buy”. Thecustomer can subsequently swipe his or her finger from one end of thebarcode 302 to the other. In response to the swipe gesture, the retailprocess manager 104 scans the barcode 302 and adds the product 300 tothe customer's shopping cart. The customer may subsequently place theproduct 300 in the cart and moves on to the next. When he or she is donewith shopping, the customer can walk out the door. The computing device100 may recognize that the customer left the store, and in responsethereto, the computing device 100 may total the transaction and chargehis or her financial account.

In another example scenario, an employee in a retail store and carryingthe computing device 100 can use a “point and swipe” gesture to takeinventory of items being placed on store shelves.

In accordance with embodiments of the present subject matter, the userinterface 106 of the computing device 100 shown in FIG. 1 may include avideo display. The retail process manager 104 may apply augmentedreality (AR) to an existing video feed from the image capture device102. The retail process manager 104 may use image recognition to findthe barcode for a product, such as the product 300 shown in FIG. 3. Inthe video display, a blue box is placed around the barcode 302 andarrows point to buttons that provide additional options. In thisexample, the customer can buy the product, obtain additional productinformation or obtain reviews about this product. The customer may touchthe buttons. When the camera picks up, the customer's hand (or finger)and the retail process manager 104 recognizes the button being touched,the button action may be implemented.

FIG. 4 illustrates a screen display 400 depicting another example imagecaptured by a computing device in accordance with embodiments of thepresent subject matter. Referring to FIG. 4, the image may be one frameof a video captured of a side of the product 300 having the barcode 302printed on a side thereof. Further, the captured video may include thehand 304 of a user of the computing device 100 moving in a variety ofdirections. It is noted that any body part or an extension thereof maybe used to indicate a user gesture.

The example of FIG. 4 is another example by which the retail processmanager 104 can determine whether a user gesture interacts with themachine-readable image in accordance with a predetermined gesture. Forexample, the retail process manager 104 may recognize that the hand 304is the hand of a user of the computing device 100. Further, the retailprocess manager 104 may recognize that the hand 304 may be making apoint gesture and moving in a swiping motion across the recognizedbarcode 302 in the direction 306. The user gesture may be deemed by theretail process manager 104 to be in accordance with a predeterminedgesture for interacting with the barcode 302. Although a swiping gestureand a point gesture is described as being recognized in this example, itshould be appreciated that any other suitable gesture captured in one ormore images may be recognized as being indicative of an interaction witha machine-readable image in accordance with the present subject matter.

In an example, the retail process manager 104 may recognize or determinethe predetermined gesture includes using one or more fingers 402 of thehand 304 of the user to gesture with respect to the machine-readableimage. A finger 402 may point to and may be recognized as a tapping 404gesture with respect to the machine-readable image. The tapping 404gesture may be determined either by a tapping-type movement (e.g., moveup and down as shown by double arrow 406 in predetermined relationshipto the machine readable code) or by an audible sound being predefined orpredetermined to indicate the tapping 404 gesture. The tapping gesture404 may include more than one tap sequence and may be gestured such asto indicate a different command relative to the barcode 302 such as add,subtract, or add more than one product 300. As an example, one (1) tapmay indicate a predetermined retail process or command of adding theproduct 300 associated with the recognized barcode 302 to a computingbased shopping cart, whereas two (2) taps may indicate the subtractionof the product 300 from the transaction. The tapping 404 gesture may bepaired with a finger 402 location gesture such that a finger 402 may berecognized on one side of the barcode 302 or within the barcode 302 suchthat if a tapping 404 gesture is detected, a command is executed whenthe command is associated with that particular location and tapping 404gesture. Additionally, tapping 404 over the top of the barcode 302 maymean to add the product 300 represented by the barcode 302 to thetransaction whereas tapping to the right of the barcode 302 may indicatethat an instruction for subtracting this product 300. Alternatively,tapping to the right of the barcode 302 may indicate an instruction toadd two of the products 300 to the transaction, as an example.

In another example, the retail process manager 104 may recognize ordetermine the predetermined gesture is that the finger 402 is pointingto and swiping under, over, or alongside of the recognized barcode 302.More than one swipe sequence may be made such as to indicate a differentcommand is to be executed relative to the product 300 such as add,subtract, or add more than one item, as an example. In another example,the retail process manager 104 may recognize or determine thepredetermined gesture includes the finger 402 pointing to and be drawinga geometric shape 408 around the code. For example, the finger 402 maybe recognized as drawings a shape such as, but not limited to, a circle,square or triangle around the code. Each geometric shaped 408 gesturemay initiate a different command to interact with the product 300 suchas add, subtract, add more than one item, or another command associatedwith the product 300. Other geometric shaped gestures 408 may be drawnto indicate different commands or instructions. It is noted thegeometric shape 408 may be substantially similar to a predeterminedgeometric shape 408 such that the retail process manager 104 mayrecognized the geometric shape 408 drawn as the geometric shape 408. Inanother example, the retail process manager 104 may recognize ordetermine the predetermined gesture involves the finger 400 pointing toand moving about in three-dimensional (3D) space (represented bycircular arrows 410) by moving toward or away from the recognizedbarcode 302 such as rotating in a circular fashion relative to thebarcode 302. It is noted that movement in 3D space 410 may be anymovement not in the same plane or a parallel plane as the barcode 302.As described herein, other commands may be executed to buy the product412, retrieve product information 414, and retrieve product reviews 416,as examples. Any executable command may also be integrated into anaugmented reality interface as described herein. As an example, the usergestures described herein may be determined to interact with anaugmented reality image of the machine readable code or with additionalPOS product information functions or commands that may be displayed inthe augmented space which may be interacted with by the user gestures asdescribed herein. In one example, a user gesture or command may bedetermined to be for returning a previously-purchased item or product.

FIG. 5 illustrates a screen display 500 of another example imagecaptured by a computing device in accordance with embodiments of thepresent subject matter. Referring to FIG. 5, the image may be one frameof a video captured of a side of the product 300 having the barcode 302printed on a side thereof. Further, the captured video may include ahand 304 of a user of the computing device 100 moving in any directiondesired.

The example of FIG. 5 is another example by which the retail processmanager 104 can determining whether a user gesture interacts with themachine-readable image in accordance with a predetermined gesture.Continuing the aforementioned example, the retail process manager 104may recognize that a finger 402 on more than one (1) hand 304 may berecognized while determining 204 whether the user gesture interacts withthe machine-readable image.

In an example, two fingers 402 come together 502 about the outside ofthe barcode 302 area to indicate a command. The command may represent aselection of the product 300 associated with the barcode 302.Alternatively, the two fingers moving in a direction apart from eachother about the outside of the barcode 302 area may indicate a de-selectcommand. As another example, more than one (1) finger 402 may becombined with any of the above gestures. As an example, the more thanone (1) finger 402 may interact geometrically (not shown), forming acircle with a thumb and a forefinger. Alternatively, two (2) hands 304may be used to form a figure relative to the barcode 302 such as thethumbs and forefingers from each of two hands 304 forming an area thatcan be placed above a barcode 302 to represent a command. As anotherexample, a particular finger 402 (e.g., an index finger) extended on onehand 304 may represent a product 300 select and the hand 304 may berotated such that the particular finger may rest adjacent to the barcode302 to execute an associated command. As another example, the user maydesire to snap the fingers 402 on the hand 304 together to indicate byaudible sound or movement a desired command. Additionally, at least onefinger 402 on one hand 304 moving relative to at least one finger 402 onanother hand 304 may be determined to indicate a command about thebarcode 302. As an example, a user may form a “U” shape with two (2)fingers 402 (e.g., an index finger and a thumb) on one hand 304 abovethe barcode 302 while the index finger 402 of the second hand motions upand down to indicate an action, such as a simulated scrolling action.The simulated scrolling action may be used to indicate an action forinteraction with an augmented reality image of the machine readable codeor with additional POS product 300 information functions. Alternatively,the user may ball the fingers 402 into a first and release in front ofor relative to the barcode 302. The user may also flash at least onefinger 402 to indicate a number of items associated with the barcode 302to put in the electronic shopping cart associated with the transaction.

FIG. 6 illustrates a screen display 600 depicting another example imagecaptured by a computing device in accordance with embodiments of thepresent subject matter. Referring to FIG. 6, the image may be one frameof a video captured of a side of the product 300 having the barcode 302printed on a side thereof. Further, the captured video may include ahand 304 of a user of the computing device 100 moving in a directionindicated by the direction arrow 306.

The example of FIG. 6 is another example by which the retail processmanager 104 can determine whether a user gesture interacts with themachine-readable image in accordance with a predetermined gesture.Continuing the aforementioned example, the retail process manager 104may recognize that one or more fingers 402 on more than one (1) hand 304may be recognized to determine that the user gesture interacts with themachine-readable image.

In an example, a finger 402 from the hand 304 may be recognized asinteracting with the barcode 302 such as an index finger 602 pointing toa barcode 302 representing a product 300 may be recognized as anindication for the computer (e.g. computing device 100 shown in FIG. 1)to display information about the product 300, such as nutritioninformation as an example. Additionally, the index finger 602 may moveto initiate an action or command as described herein. Alternatively, theindex finger 602 may remain substantially motionless for a predeterminedperiod of time, such as a few seconds (e.g., two (2) seconds). Asanother example, the index finger 602 and an extended thumb 604 mayindicate a computer action such as retrieval of the product 300 price.Another example, may be to extend the index finger 602 and also themiddle finger 606 to indicate the desire to retrieve productmanufacturing. Other examples may be to extend any combination of none,one (1) or all of the fingers on any number of hands to indicate anaction or command as desired. It is noted that in response todetermining that the user gestures as described herein interact with themachine-readable image in accordance with the predetermined gesture, apredetermined retail process in association with the product 300 may beexecuted or implemented.

Disclosed herein are several examples by which it is determined that auser hand gesture interacts with a machine-readable image in accordancewith a predetermined gesture for implementing a predetermined retailprocess. It should be understood that any number and combination ofgestures may be recognized as being in accordance with a predeterminedgesture. Such gestures may be recorded or otherwise captured by one ormore image capture devices. Subsequently, the captured image(s) may beprocessed and analyzed for determined whether there is interaction withthe machine-readable image in accordance with a predetermined gesture.

In accordance with embodiments, predetermined gestures may be combinedwith one or more other user inputs for determining whether to implementa predetermined retail process. For example, a user may make a soundand/or move a computing device for implementing a desired retailprocess. In an example, the user may speak one or more words and makeone or more user gestures (e.g., tapping on a product barcode). Togethersuch inputs may be recognized as being a command for implementing apredetermined retail process. In a particular example, a user may“swipe” his or her finger across a barcode and speak “add three” foradding 3 of the products to a purchase transaction or inventory count.The other input may be any other suitable user input that may berecognized by a computing device. As an example, the user input may beinput into a user interface, such as a keypad or touchscreen display.

The present subject matter may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent subject matter.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present subject matter may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Java, Smalltalk, C++ or the like,and conventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present subject matter.

Aspects of the present subject matter are described herein withreference to flowchart illustrations and/or block diagrams of methods,apparatus (systems), and computer program products according toembodiments of the subject matter. It will be understood that each blockof the flowchart illustrations and/or block diagrams, and combinationsof blocks in the flowchart illustrations and/or block diagrams, can beimplemented by computer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present subject matter. In this regard, each block inthe flowchart or block diagrams may represent a module, segment, orportion of instructions, which comprises one or more executableinstructions for implementing the specified logical function(s). In somealternative implementations, the functions noted in the block may occurout of the order noted in the figures. For example, two blocks shown insuccession may, in fact, be executed substantially concurrently, or theblocks may sometimes be executed in the reverse order, depending uponthe functionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

While the embodiments have been described in connection with the variousembodiments of the various figures, it is to be understood that othersimilar embodiments may be used or modifications and additions may bemade to the described embodiment for performing the same functionwithout deviating therefrom. Therefore, the disclosed embodiments shouldnot be limited to any single embodiment, but rather should be construedin breadth and scope in accordance with the appended claims.

What is claimed:
 1. A method comprising: capturing, using an image capture device, one or more images including a machine-readable image and a user hand gesture; identifying the machine-readable image as being associated with a product; integrating the identified machine-readable image and the user hand gesture into an augmented reality interface; identifying the user hand gesture as it interacts with the machine-readable image in a three-dimensional (3D) space, wherein the user hand gesture interaction occurs in the three dimensional space outside a same plane as the machine-readable image within the augmented reality interface; determining whether the identified user hand gesture in 3D space is in accordance with a predetermined gesture; and in response to determining that the identified user hand gesture in 3D space is in accordance with the predetermined gesture, executing a predetermined retail process in association with the product.
 2. The method of claim 1, wherein the machine-readable image is one of a barcode, a two-dimensional code, a three-dimensional code, a portion of the product, and a product label, and wherein the machine-readable image represents data that identifies the product, and wherein identifying the machine-readable image comprises interpreting the data to identify the product.
 3. The method of claim 1, wherein the predetermined gesture comprises one of a swipe gesture and a point gesture with respect to the machine-readable image.
 4. The method of claim 1, wherein the predetermined gesture comprises one or more fingers of a user gesturing with respect to the machine-readable image.
 5. The method of claim 1, wherein the predetermined gesture comprises a tapping of the one or more fingers with respect to the machine-readable image.
 6. The method of claim 1, wherein the predetermined gesture comprises drawing a geometric shape with the one or more fingers with respect to the machine-readable image.
 7. The method of claim 1, wherein the predetermined gesture comprises a three-dimensional movement gesture with respect to the machine-readable image.
 8. The method of claim 1, wherein the predetermined gesture comprises determining one or more fingers of a first hand and one or more fingers of a second hand gesturing with respect to the machine-readable image.
 9. The method of claim 1, wherein the predetermined gesture comprises one or more fingers on a first hand moving relative to one or more fingers on a second hand for gesturing with respect to the machine-readable image.
 10. The method of claim 1, further comprising determining predetermined user input into a computing device, and wherein implementing the predetermined retail process comprises implementing the predetermined retail process based on the predetermined user input.
 11. The method of claim 10, wherein the predetermined user input includes one or more of sound, another user gesture, and user input.
 12. A computing device comprising: an image capture device configured to capture one or more images including a machine-readable image and a user hand gesture; and a retail process manager including at least a processor and memory configured to: identify the machine-readable image as being associated with a product; integrate the identified machine-readable image and the user hand gesture into an augmented reality interface; identify the user hand gesture as it interacts with the machine-readable image in a three-dimensional (3D) space, wherein the user hand gesture interaction occurs in the three dimensional space outside a same plane as the machine-readable image within the augmented reality interface; determine whether the identified user hand gesture is in accordance with a predetermined gesture; and execute a predetermined retail process in association with the product in response to determining that the user hand gesture is in accordance with the predetermined gesture.
 13. The computing device of claim 12, wherein the machine-readable image is one of a barcode, a two-dimensional code, a three-dimensional code, a portion of the product, and a product label, and wherein the machine-readable image represents data that identifies the product, and wherein the retail process manager is configured to interpret the data to identify the product.
 14. The computing device of claim 12, wherein the retail process manager is configured to determine one of a swipe gesture and a point gesture with respect to the machine-readable image.
 15. The computing device of claim 12, wherein the retail process manager is configured to determine whether one or more fingers of one or more hands of a user gesture with respect to the machine-readable image.
 16. The computing device of claim 12, wherein the retail process manager is configured to determine a tapping of the one or more fingers with respect to the machine-readable image.
 17. The computing device of claim 12, wherein the retail process manager is configured to determine a drawing of a geometric shape with the one or more fingers with respect to the machine-readable image.
 18. The computing device of claim 12, wherein the retail process manager is configured to determine a three-dimensional movement gesture with respect to the machine-readable image.
 19. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions readable by a computing device to cause the computing device to: receive, by the computing device, one or more captured images including a machine-readable image and a user hand gesture; identify, by the computing device, the machine-readable image as being associated with a product; integrate the identified machine-readable image and the user hand gesture into an augmented reality interface; identify the user hand gesture as it interacts with the machine-readable image in a three-dimensional (3D) space, wherein the user hand gesture interaction occurs in the three dimensional space outside a same plane as the machine-readable image within the augmented reality interface; determine whether the identified user hand gesture is in accordance with a predetermined gesture; and execute a predetermined retail process in association with the product in response to determining that the user hand gesture is in accordance with the predetermined gesture. 